Digital Watermarking Using Deep Neural Network

Recently in the vast advancement of Artificial Intelligence, Machine learning and Deep Neural Network (DNN) driven us to the robust applications. Such as Image processing, speech recognition, and natural language processing, DNN Algorithms has succeeded in many drawbacks; especially the trained DNN models have made easy to the researchers to produce state-of-art results. However, sharing these trained models are always a challenging task, i.e. security, and protection. We performed extensive experiments to present some analysis of watermark in DNN. We proposed a DNN model for Digital watermarking which investigate the intellectual property of Deep Neural Network, Embedding watermarks, and owner verification. This model can generate the watermarks to deal with possible attacks (fine-tuning and train to embed). This approach is tested on the standard dataset. Hence this model is robust to above counter-watermark attacks. Our model accurately and instantly verifies the ownership of all the remotely expanded deep learning models without affecting the model accuracy for standard information data.


I. INTRODUCTION
Deep learning and machine learning methods have recently become a hot topic in many computers vision tasks, such as objects and images recognition in still images.
While successful techniques have been manifested with image understanding, video content still presents additional challenges e.g., motion, temporal consistency, the spatial location that usually cannot be bridged with still image recognition solutions. In the structure of DNN neurons are interconnected, and edges are joined together [1], [2]. In DNN we solve the complex set of calculations by using input, convolution, pooling and out layers [3]. Deep Learning wrapper facilitates the design and training of models over many frameworks such as Theano, Tensor Flow, Torch, Chainer, and Keras [4]- [8]. These frameworks reduce the efforts for engineer and researcher in deep learning but still the training the deep model is not easy so it acquires the massive amount of data Manuscript  and time. DNN found in multiple applications, GoogleNet, ResNet, VGGNet, AlexNet and LeNet [9]- [13]. This paper focused on the classification of the watermark image using DNN.
Few models take several days to train deep models such as ResNet and VGGNet [10], [11]. So, therefore, it is difficult to train the model for this problems nowadays GPUs are commonly used to save the training and computation times. Also, some training models are online available on the website which offers comfortable help to try these models directly. Hence sharing models is always an essential part in research progress and future development of deep neural networking (DNN). Other digital platforms offer trained models similar to google app store or even artificial intelligence models available, but security is required to protect these models.
In our work, we employed DNN models for digital watermarking Technology. Watermarking scheme for deep neural networks is proposed which is the black box in terms of verification. Watermarking is applied to recognize the possession of the copyright of digital contents such as audios, images, and videos.
We perfumed the experiment watermark embedding through DNN, in the watermarking process, training the neural network to be as accurate as possible concerning these randomly chosen layers and neural network formalize the idea of backdooring a neural network with specific properties. People prefer neural networks now because parameters are not as important as such as the performance. Hence, it is necessary to improve the host trained network performance. There are some main properties which we want to achieve with our watermarking scheme the first one is it supposed to be functionality preserving there's no point of watermarking. Functionality preserving so does a watermark harm the task that you want to achieve it turns out so we ran experiments training without the watermark and with the watermark and it turns out that the accuracy of the neural network is almost the same like it sometimes even increases a bit.

II. DNN WATERMARKING
Digital watermarking is the process to protect the hidden information into the digital media to preserve the ownership of those media data. Many approaches have been introduced to obtain the watermark to be active as well as robust to removal of different attacks. Based on extraction methods, digital image watermarking approaches can be split into two main categories named Blind-watermarking and Non-Blind watermarking [14]. Traditional algorithms for digital watermarking can be classified into two methods transform domain also well known as frequency domain [15] and spatial domain [16]. Blind watermarking only need Digital Watermarking Using Deep Neural Network Farah Deeba, She Kun, Fayaz Ali Dharejo, Hameer Langah, and Hira Memon watermarked images for the extraction process; however, in non-blind watermark approach watermarked images, original images, and extra information is also required to extract information. Spatial domain digital watermarking algorithms have been proposed [16]- [18].
Uchida et al. [19] previously introduced the first approach for embedding watermarks into a deep neural network through which he embedded the data into the weights of DNN. Hence, it believes that the stolen prototypes can be nearby accessible to obtain the group of parameters, which is not possible since most deep learning models are used as online assistance and it will be troublesome to immediately procure access to model parameters, particularly for the stolen prototypes. As DNN models are broadly deployed and become more critical, they are frequently targeted by enemies. Enemies can steal the model and build a copy of AI service.
There are two types of DNN watermarks that are used mainly, white box and the black box. In the white box, watermark techniques have advantages over black box watermarking techniques in terms of operator information. However, the ability to transmit a lot of information has the cost of limited application scenarios, since it requires that internal models are publicly available for the extraction of WM. The bit error rate (BER) between the extracted WM and the background truth must be zero to demonstrate the authorship of the model consulted.

A. White-Box Watermarking
Uchida et al. [20] first introduced the DNN watermarking scheme, which contains a multi-bit WM in the weighting of selected layers of the target DNN by adding an additive binary cross-entropy loss as a regularization member of WM. When training a DNN model, a watermark is coded in the order of weights, losing the personalized regularization, thereby penalizing the difference between the desired WM and the transformation of the weights. With the selected transformation weights, the WM is extracted from the consulted model. It is robust to fine-tuning and compression attacks on models also maintain the accuracy of the model.
Rouhani et al. [4] presented a general watermarking framework that works in both the white box and the black box environment. The embedded bit watermark is embedded in the activation cards of the selected layer (s) of the target DNN by integrating two additional regularization loss terms, binary cross-entropy loss, and GMM agent loss. To activate cards, a deep Gaussian Mixed Model (GMM) is used. During DNN training, WM-specific regularization loss conditions are integrated to match the activations and encode the WM information. Later in the extraction step, the WM key frames are transferred to the model to trigger a marked activation. The BER is calculated and the result of these activation watermarks is restored. This has an advantage over the Uchida [6] because it gives both data and model-dependent. At the same time, it is robust against attacks with fine-tuning, overwriting and parameter setting. Since it is both data and model-dependent but more computation.

B. Black-Box Watermarking
There are several methods for the black box watermark [21]- [24] that target the zero bit. The presence of the WM is determined by interrogating the model with the WM key frames and determining the appropriate accuracy by thresholding. Black boxing WMs are more practical in the real world as they are more relaxed, instead of API access rather than model internals, but with limited capacity.
Merrer et al. [22] suggest creating controversial patterns if the opponent succeeded in the attack, the corresponding samples are called "true opponents". In this scheme, the set of opposite samples is used as the WM key set to change the decision boundary of the target neural network. While attacks fail, images l watermark key is a complete combination of true and false opponents. The model is queried in the detection phase of watermark and the null hypothesis is tested, this is done via the Binary distribution. If the watermark key is less than the threshold, the watermark should be present in the model. It is practical from the point of view of detection and embedding, but the accuracy can be lowered after embedding watermarks, achieves a very high false-positive rate and is not very robust against attacks. Images incorrectly classified by the model are suggested by Yossi et al. [1]. These misclassified images known as backdoor images and random labels as a watermark key that is set to embed the watermark with a zero bit that is DNN compliant. The comparable basis of the key label is randomly selected from all classes except the real label and the predictable real sample. First, the threshold is set by the binomial distribution and then compared to the watermark trigger set to detect the watermark. This scheme provides better accuracy and a high detection rate but is sensitive to water removal attacks. Zhang et al. [24] introduced the three different key generation schemes content-based, noise-based and unrelated-based images respectively. Pre-train model is accurately adjusted with watermark keys and in the detection phase, the user sends the watermark key to the DNN model. This is the external service provider that ultimately submits the threshold to classify the Boolean decision. He introduced three different methods for generating a watermark key, but not well for giving a continuous performance on the standard datasets. Rouhani et al. [23] in the goal, DNN investigated the rarely inhabited space to regard the random images and labels as a watermark set. The watermark key used for identifying the key pairs and for Watermark embedding fine-tuning the target DNN model. The final WM key set is the intersection of the keys that are correctly predicted by the highlighted model and incorrectly predicted by the unmarked model. In the multinomial distribution of the detection phase for the output prediction and does the examination of the imaginary hypothesis with the definitive Watermark key set. It gives a high detection rate with a low percentage of false alarms but the generation of keys generates higher overhead costs. Look at fake enemies that preserve the model's accuracy for specific data.
As DNN can learn the patterns automatically once the parameters are fixed; hence we implemented DNN in watermarking one is to protect the Intellectual properties and second is to verify the ownership with embedded watermarking. The goal of the framework is to preserve the intellectual characteristics of the deep neural networks through verifying possession of remote DNN services with embedded watermarks. In our model first, we defined the labels for different watermarks and also trained these labels to the deep neural network. In the model without the watermark by using less computation a lot less estimate than training from scratch. It can never defend against an enemy who makes a model related to cryptography where we assume that there is a gap in computational power between the attacker and the defender.
The Diagram of proposed model is given in Fig. 1. From Fig. 1 we can recognize the complete typical watermarking process. An embedding algorithm "E" embeds the pre-trained watermarks W into the carrier data C that should be protected from attacks. When embedding is done the data stored in (e=E(W,C)) where decryption is appliedto extract the watermark "W" from e. The above Fig. 2 shows the DNN watermark Framework. When generating Watermarks these watermark shown as the fingerprint in ownership as well. It first creates the watermark; these watermarks reported as the fingerprint verification. Then these watermarks embed to DNNs through training. The DNN automatically learns the patterns of watermarks and retains them. And finally for verification after embedding our models is strong enough to verify the ownership. Following newly generated models able to verify ownership so that the owner can prove by simple sending watermark as input and checking through output by deploying DNN model. Rest our paper contains Watermark Generations, types of attacks (fine-tuning and train to embed) and owner verification and finally we demonstrated the Experiment Setup.

III. EMBEDDING AND WATERMARKS GENERATION
In embedding retraining to input distribution which is entirely unrelated to the task on the training data using the labels in the neural network as accurate on the strain data as possible without over-fitting. At the same time these random inputs which chose independently of the input distribution. Accept arbitrary labels first fix and then during different ways how to determine the input distribution they use different security they have different security properties they want to show for the neural network and therefore have a separate analysis We applied the watermark algorithm which already used by Zhang et al. [14].

A. Algorithm: Watermark Embedding
The algorithms of embedding show that DNN Watermark embedding performs the original training data (Dtrain) and transformation keys {Ys, Yd} (s, d) which are inputs and finally outputs protected the DNN model (Z8, Dwm). With the help of transform key, we label the watermarks which are defined by the owner. Here Yd is owner defined labels for the watermark and Ys is original label data.
We took the same amount of training with and without the embedding watermark so that it probably increases a bit is like apparently just outliers. We also validated and experimentally verify that it is not only able to remove the watermark on one specific attack, but there are a few more attacks. so we did a fine-tuned either only the last layer or all layers , as fine-tuning is a standard thing to do if you have more training data to adapt your neural network. The following two circumstances noticed that the host network of the copyright owner is supposed to embed a watermark to the host network in training or fine-tuning. We Recognized watermarks, the histogram of the embedded watermark and without watermarks through σ (ΣiXjiwi) as shown in Fig. 3.
In the first image we see only the image without watermarks (only the blue histogram), and in the second image the watermark is used (with the red and blue histogram).

B. Train-to-Embed
Train to embed means the training from scratch and embedding the watermark where labels data training available of the host network.

C. Fine-Tuning to Embed
In this attack, the watermarked model is re-trained with the traditional cross-entropy loss of the original training data set. A robust watermark must be able to change weights during model fine-tuning. Fine-tune the whole network which means to annihilate the watermark this was kind of unexpected for us because at the same time embed the watermark during the training.
When operating the entire dataset, we split the test dataset into two parts. In the first part, fine-tuning is used for previously trained DNN and in the second part for the evaluation of additional models. Later, testing and watermarking several new models to estimate the robustness of our watermark frame for fine-tuned changes.

D. Ownership Verification
To verify the ownership, we must send the regular queries with generated watermarks images that is Dwm, if the Query matches which means xwm=Ywm we can validate that our model is protected. It is due to the DNN model [20]. Because without embedding watermarks cannot understand watermarks so that it can classify images but will not classify images with embedded watermarks.

E. Meaningful Content (Watermark-Content)
In training images, we add the text to give useful content on it; these images taken as input images. For example, if we insert the "TEST" string in our model to the Lena image. In this example, the query watermark (TEST) in the Lena image as a predefined prediction (Lena) which consist of fingerprints for model verification. Fig.4 shows the Lena image with and without content.

F. Overview of Proposed Matrix Image
During training on the different images, Random Matrix Image auto-generated. The matrix can be generated using the random function with the randomized number from the specified range. This matrix gives shows the output of our proposed DNN model. The auto-generated Matrix of the Lena image is shown in

V. EXPERIMENT
We have done all our experiments by using Python 2.7 with Spyder Integrated Development Environment(IDE) and Keras [8] and TensorFlow [5] frameworks using device GforceGTX 770, CUDNN 5110. We assess the performance of our watermarking framework with the standard dataset with learning ratee of 0.5. We trained our model on multiple epochs. Each time it gives almost the same error rate and the same accuracy. So we notice that our embedding DNN model perform well against the types of attacks which means it can embed without the impairing the performance of the host network. Here Table I and Fig. 7 show the accuracy of our model during Training. Table II shows the Architecture of our DNN model.
We have inserted a watermark in the following we will mention the location of the host layer by simply drawing the convolution 2, 3 and 4 groups.   Fig. 8 shows the histogram of the watermark marks detected. There are two points, the red and the blue, the red screws the histogram from embedded while the blue is not marked embedded. From the Fig. 8, we can see that much close-up case that helps to detect the presence of watermark known as a zero bit watermark.

VI. CONCLUSION AND FUTURE WORK
We generalized the digital watermark using the deep neural network, and remotely verify the ownership of DNN models based on the embedded watermarks. Our paper explained the idea of the digital watermark in Deep Neural Network. We learn different frameworks of watermarking, embed them into DNN model and finally, we perform owner verification. Achievement of this DNN model successfully verified owner verification based on embedded watermarks. These all performance evaluated on the Standard Dataset, and we show that our framework can meet the general watermarking standard; hence our model is robust to above possible attack.
Our model gives Inconsistent performance across different benchmarks so in future still the accuracy can be improved and also can be tested on different Stoddard datasets such as CIFAR-10, CIFAR-100.

CONFLICT OF INTEREST
"The authors declare no conflict of interest".

AUTHOR CONTRIBUTIONS
Farah Deeb and FA Dharejo has produced the results and write the manuscript. She Kun has guided in the experiments and supports in funding. While Hameer and Hira have analysed data and all results.