3.1.

Alps Dataset

We designed the Alps dataset to train and test the architecture for the season transfer manipulation. The alpine area, in fact, is characterized by marked differences between winter and summer, with winter images largely covered by snow, and summer images containing large areas of green vegetation. For this dataset, we only selected the RGB and NIR bands, with ground sampling distance (GSD) equal to 10 m, for a total size of $\mathrm{10,980}\times \mathrm{10,980}$ resolution and 16 bits per pixel. We collected images representing exactly the same area taken at two different months, each month representing a different season. In this way, in addition to using the images for training the season transfer architecture, we also have a way to compare the results of the season transfer with ground-truth images. In particular, we selected images taken in June 2019 for the summer dataset and in December 2019 for the winter dataset.

The description of the procedure we followed to build the dataset is described in the following. To start, we selected only images with limited cloud coverage. Since it was not possible to get only images with 0% cloud coverage, we limited the search to images with a cloud coverage lower than 9%. We extracted the bands of interest (RGB and NIR) from the downloaded products as jp2000 images. We used the gdal software library³⁷ for reading and writing raster and vector geospatial data formats. Specifically, we used the gdal retile command to tile the downloaded images from their initial size into several $512\times 512$ images. Then we removed the edge tiles. Finally, we paired the images of areas existing in both the winter and summer domains. As a result, we built a dataset of 3936 pairs of images with $512\times 512$ resolution. In Fig. 1, we show the RGB version of some sample images of the Alps dataset.

Fig. 1

Example images from the datasets: (a) Alps pairs, (b) Scandinavian pairs, (c) land cover images, and (d) SEN12MS images.

3.2.

Scandinavian Dataset

To validate the effectiveness of the season transfer architecture on different kinds of landscapes, we built another dataset with marked differences between winter and summer. The dataset includes images from Scandinavia and was built similarly to the Alps dataset. In this case, the selected date range includes June 2020 for the summer and February 2020 for the winter. For both domains, we selected only images with 0% cloud coverage. The final dataset consists of 9000 paired images of size $512\times 512$. Figure 1 shows some RGB examples of the images from the Scandinavian dataset.

3.3.

Land Cover Datasets

The second kind of global manipulation we have considered aims at transforming barren covers into vegetation areas and vice versa. For this reason, we created the land cover dataset by selecting images, in which one of the two types of covers is predominant with respect to the other. To do so, we first selected the areas of interest based on the statistics provided by the organization for economic co-operation and development.³⁸ With regard to the areas with a high percentage of vegetation, we extracted images from Congo, Salvador, Montenegro, Gabon, and Guyana. The cloud coverage was set to 0%, and the range of dates spanned from June 2019 till December 2019. For the barren domain, we selected the areas of interest in South and Central America. For both domains, we used a linear discriminant analysis classifier³⁹ to make sure that after cropping the images to $512\times 512$ patches, they contain, respectively, a great percentage of vegetation and barren soil. For each domain, we collected 10,000 images. For the vegetation domain, the average percentage of vegetation pixels in an image is 98% with a maximum of 100% and a minimum of 60%. For the barren domain, the average percentage of barren pixels in an image is 82% with a maximum of 100% and a minimum of 63%. In Fig. 1, we show some RGB examples for the two different domains.

3.4.

SEN12MS

The SEN12MS²⁴ dataset was created to provide a large-scale satellite dataset for developing DL-based methods. As opposed to the previous datasets, SEN12MS has a larger variety of images regarding spatial coverage, diversity, and number of available samples. The dataset contains 180,662 triplets of multispectral patches, dual-pol SAR image patches, and MODIS land cover maps collected from Sentinel-1 and Sentinel-2 satellite imagery. The images span all seasons with an approximately equal number of images captured during winter, spring, summer, and fall. The images’ locations vary with respect to sea level elevation, climate, latitude, and urbanization level. Each patch has a resolution of $256\times 256$ pixels and we only used the RGB channels of the multispectral patches. In our experiments, we used images from Africa, Europe, Asia, Australia, and South America, for a total of $\sim 120,000$ images. Figure 1 shows some RGB images from SEN12MS.

3.5.

World Dataset

Eventually, we collected images from various regions in the world by downloading them from the Copernicus Hub to construct the world dataset. The images were captured in 2018 and have an equal number of images across seasons. We only used the RGB bands, which are sampled at 10-m GSD with image size equal to $\mathrm{10,980}\times \mathrm{10,980}$. The images can contain clouds up to 2% of the total number of pixels. From these images, we extracted nonoverlapping patches of size $512\times 512$. The world dataset contains 285,768 images. We used the world dataset images to generate images containing spliced objects as described in the subsequent sections. Figure 2 shows four examples of world dataset images.

Fig. 2

Example of images from the world dataset.

4.1.

Season Transfer

As we said, with regard to global manipulation, we considered two different objectives. The first objective was to transfer images taken in the winter to their summer counterpart and vice versa. Paired images, with synchronized input and ground truth images, are not difficult to get in this case since the same location is usually available for download in both seasons. For this reason, and since using paired images facilitates training, we used the pix2pix architecture.¹⁴ Pix2pix is a variant of traditional GANs, where instead of generating an image from noise, a conditional GAN (cGAN)⁴⁰ is considered. The cGAN takes one image as input and translates it into an image belonging to the target domain. The training workflow of the pix2pix is shown in Fig. 3. The generator attempts to generate a winter image ($x$) starting from its summer counterpart ($x^{\prime }$) and vice versa. The goal of the discriminator is similar to that of a classical GAN, namely to judge if an image is a real or a synthetic one. As opposed to classical GANs, however, the discriminator takes as input a pair of images, an original image from the source season, and an image depicting the same area in the target season, the goal being to decide if the image of the target season is real or fake. A model can be trained to transfer images in one direction only, which is from winter to summer or from summer to winter, so to be able to apply bidirectional transfers, we had to train two models.

Fig. 3

Pix2pix training workflow.

4.2.

Land Cover Transfer

The second global manipulation we have considered is land cover transfer, whereby barren images are transferred to vegetation images and vice versa. A noticeable difference with respect to season transfer is that, in this case, we have no ground truth images since in most cases the barren (res. vegetation) version of a vegetation (res. barren) image does not exist. This requires that a different architecture and training strategy be used. In particular, we opted for the CycleGAN architecture,¹⁶ which, unlike pix2pix, does not require the availability of paired images for training.

The CycleGAN architecture consists of two generators and two discriminators. Let us assume that the goal of the CycleGAN is to transfer images from a domain A to a domain B and vice versa. Each generator translates the images in one direction. Specifically, the first generator transfers images from domain A to domain B, whereas the second generator transforms the images in the opposite direction. In this way, each generator can act as an additional constraint for the other. The basic idea behind CycleGAN is to enforce a cyclic consistency in such a way that when the output of the first generator is used as an input for the second, the image produced by the second generator should be as close as possible to the original input image (thus avoiding the need for paired images belonging to the two domains). Note that, unlike with pix2pix, it is not necessary to carry out two separate training for the two directions of the transfer since the two generators that are part of cycleGAN architecture provide the models for the two directions.

4.2.1.

Architecture

The exact architecture we have used to implement the land cover cycleGAN is described in the following. The generators are implemented by means of a residual network⁴³ with six residual blocks and skip connections. The input size we used is $512\times 512\times 4$. Each residual block has a convolutional layer, a batch normalization layer, and a leaky ReLU activation function layer. Regarding the discriminator, we used seven convolutional layers, each followed by batch normalization and a leaking ReLU activation. For both networks, Adam’s optimizer was used. Figure 4 shows the different losses of a cycleGAN architecture. Training is obtained by finding a good trade-off between three partial losses. The first partial loss is the typical adversarial GAN cross entropy loss ($L_{\mathrm{adv}}$) defined as

Eq. (4)

$$L_{\mathrm{adv}}=E_v[(D_b(G_{v2b}(v))-1{)}^2]+E_b[(D_v(G_{b2v}(b))-1{)}^2],$$

where $G_{v2b}$ is the generator that translates the images from vegetation to barren, $G_{b2v}$ is the generator that transfers the images from barren to vegetation, $D_b$ and $D_v$ are the discriminator networks that classify images as real or fake for the barren and vegetation domains, respectively, and $b$ and $v$ are generic barren and vegetation input images. The second loss is the cyclic consistency loss defined by

Eq. (5)

$$L_{\mathrm{cycle}}=E_v[\Vert G_{b2v}(G_{v2b}(v))-v\Vert ]+E_b[\Vert G_{v2b}(G_{b2v}(b))-b\Vert ],$$

whose goal is to make sure that vegetation (res. barren) images that are translated to barren (res. vegetation) and then back to vegetation (res. barren) are as close as possible to the original input.

Fig. 4

CycleGAN partial losses: (a) adversarial, (b) cyclic, and (c) identity losses.

Finally, an extra constraint is added to ensure that when a generator is fed with an image belonging to the output domain, it leaves the image as is since no transformation is actually necessary. Such a goal is achieved by defining a third loss, namely the identity loss, as follows:

Eq. (6)

$$L_{\mathrm{identity}}=E_v[\Vert G_{b2v}(v)-v\Vert ]+E_b[\Vert G_{v2b}(b)-b\Vert ].$$

The goal of the generators is to minimize an overall loss combining the three partial losses described above:

Eq. (7)

$$\underset{G_{v2b},G_{b2v}}{\min }{\alpha }_1{L}_{\mathrm{adv}}+{\alpha }_2{L}_{\mathrm{cycle}}+{\alpha }_3{L}_{\mathrm{identity}},$$

where ${\alpha }_1$, ${\alpha }_2$, and ${\alpha }_3$ are the weights of the adversarial, cycle, and identity losses, respectively. In contrast, the discriminators aim at distinguishing between real and synthetic images, each in its domain, a goal that is achieved by solving the following optimization problem:

Eq. (8)

$$\underset{D_v,D_b}{\max }\hspace{0.17em}{E}_b[(D_b(b)-1{)}^2]+E_v[(D_b(G_{b2v}(v)){)}^2]+E_v[(D_v(v)-1{)}^2]+E_b[(D_v(G_{v2b}(b)){)}^2].$$

4.3.

Splicing with iGPT

The next manipulation we considered is local splicing. We used iGPT to generate synthetic image splices that then we inserted into images from the world dataset. The iGPT was trained on the SEN12MS dataset. The goal in this case was to remove some parts of an image from the world dataset and replace them with content generated by iGPT, by paying attention to enforce the consistency of the spliced patch with the surrounding of the removed part and the rest of the image. The overall splicing process is depicted in Fig. 5.

Fig. 5

Spliced images created using iGPT.

IGPT¹⁹ is a transformer-based unsupervised image classification and image generation model. Early transformer-based models, such as BERT,⁴⁴ RoBERTa,⁴⁵ and T5,⁴⁶ could be used directly with 1D sequences in any form, but were not easily extendable to 2D data, such as images. A model that is able to work with 2D data, namely the GPT-2 ⁴⁷ model, has been introduced recently (a GPT-2 model trained on images is known as iGPT). To identify the spliced areas, we apply watershed⁴⁸ unsupervised segmentation to the images of the world dataset and randomly select several of the largest segments as the regions where the splices generated by iGPT had to be inserted. Then we use iGPT to generate the to-be-inserted splices based on the mask given by the watershed segmentation.

4.3.1.

Architecture

IGPT¹⁹ is very similar to the GPT-2.⁴⁷ An important difference between the two architectures is the activation function. Specifically, a quick Gaussian error linear unit (GELU) is used in iGPT, instead of the GELU used in GPT-2. GELU combines some useful features of the most commonly used activation functions. It randomly multiplies its input by one and randomly sets some of the activations to zero. It can be approximated by the following equation:

Eq. (9)

$$\mathrm{GELU}(x)\sim 0.5x(1+\tanh (\sqrt{2/\pi }(x+0.044715x^3))).$$

Quick GELU is similar to GELU but with a lower computationally complexity, being defined as

Eq. (10)

$$\mathrm{quick}\_\mathrm{GELU}(x)=\frac{x}{1+e^{-1.702x}}.$$

IGPT also differs from GPT 2 in the number of normalization layers, which is much lower for iGPT, thus decreasing significantly the number of operations. The reader may refer to Ref. 19 for more details. The input of GPT-2 is a sequence of pixels $U=x_1,\dots ,x_n$. Although its objective is to maximize the following likelihood:

Eq. (11)

$$L_1(U)=\sum _i\log P(x_i|x_{i-k},\dots ,x_{i-1},\mathrm{\Theta }),$$

where $P$ is the conditional probability modeled by a neural network with parameters $\mathrm{\Theta }$, and $k$ is the size of the context window.

The transformer inside the iGPT creates a model of the probability density function of the current pixel $x_i$, given the previously observed pixels $x_1,\dots ,x_{i-1}$ as shown in the following equation:

Eq. (12)

$$p(x)=\prod _{i=1}^{n}p(x_{{\pi }_i}|x_{{\pi }_{i-k}},\dots ,x_{{\pi }_{i-1}};\theta ),$$

where in general, ${\pi }_i$ indicates a permutation of the pixel sequence. In our case, we simply let ${\pi }_i$ be the identity permutation, that is ${\pi }_i=i$. ¹⁹ $\theta$ contains all the other parameters of the neural network used during training. The 2D image is transformed into a 1D sequence by lexicographical ordering.

The input of the decoder part of the transformer is a sequence of discrete pixels $x_1,\dots ,x_{i-1}$, and the output is a $d$-dimensional vector as shown in Fig. 6. The decoder is implemented by a stack of $L$ blocks, in which the $l$’th block produces an intermediate embedded vector $h_1^l,\dots ,h_d^l$. IGPT uses the same formulation as GPT-2 for the transformer decoder block, in which we input $h^l$ in the order seen in Eqs. (1315) to obtain $h^{l+1}$:

Eq. (13)

$$n^l=\mathrm{layer}\_\mathrm{norm}(h^l),$$

Eq. (14)

$$a^l=h^l+\mathrm{multihead}\_\mathrm{attention}(n^l),$$

Eq. (15)

$$h^{l+1}=a^l+\mathrm{mlp}(\mathrm{layer}\_\mathrm{norm}(n^l)).$$

The final layer of the transformer decoder is followed by a normalization layer and projection logits (real numbers that with unlimited range) as a parameter for the conditional probability distributions of each sequence element. In the final step, the output vector is reshaped into a 2D image. In our implementation, we used eight layers to construct the iGPT with two heads, we also set the embedded vector dimension to 16, as suggested in the original iGPT paper.¹⁹

Fig. 6

Block diagram of iGPT.

4.4.

Splicing with Vision Transformer

The vision transformer¹⁷ was originally developed for image classification by replacing the convolution layers with a transformer model.⁴⁹ Specifically, transformers tend to have inductive bias when trained on large-scale datasets. The vision transformer aims at resolving this issue and provides good results for image classification by scaling the dataset to a smaller size and reducing the amount of training data. In our work, we modified the vision transformer so to use it for synthetic splice generation. In particular, we edited the last layers of the vision transformer, so to generate an image instead of a classification score. The modified layer has a size of $3\times 256\times 256$.

We use the modified vision transformer to generate synthetic image splices to be inserted into images from the world dataset. The modified vision transformer was trained on the SEN12MS dataset. The goal here is to remove some regions of an image from the world dataset and replace them with content generated by the modified vision transformer. The content should be consistent with the surrounding of the removed part and the rest of the image. Similar to iGPT, we use watershed⁴⁸ segmentation on images in the world dataset and randomly selected several of the largest segments to insert the splices. The whole procedure is shown in Fig. 7.

Fig. 7

Spliced images created using vision transformer.

4.4.1.

Architecture

The vision transformer inputs are image patches as shown in Fig. 8. In our technique, the original image size is $128\times 128$, whereas the size of the patches inside the transformer is $64\times 64$. Then the dimensionality of the input image patches is reduced using a linear projection: $T_i=W{\widehat{I}}_i$, where $W\in {\mathbb{R}}^{D\times N}$ is a linear mapping function learned during training, ${\widehat{I}}_i\in {\mathbb{R}}^N$ is the flattened $i$’th image patch, and $T_i\in {\mathbb{R}}^D$ is known as the $i$’th image token. Therefore, an image token is a linear projection of the input patches.

Fig. 8

Block diagram of the vision transformer.

The vision transformer incorporates a traditional transformer used on one-dimensional sequences. The transformers use self-attention modules to incorporate long range information, which can contain information about all the inputs,⁴⁹ let us call these inputs “position-aware tokens.” The position-aware-tokens indicate where the image patch is located in the input image. The self-attention modules are invariant to the order of the position-aware tokens, so the order with which we input the list of “position-aware tokens” into the transformer is irrelevant. We create the position-aware tokens by concatenating the image token (input image patch projection) and the positional embeddings. The positional embeddings incorporate positional information for each different input image token.¹⁷ We created these positional embeddings by numbering the order of the patches with $64\times 64$ size. These positional embeddings, $P_i\in {\mathbb{R}}^D$ for $i\in \{\mathrm{0,1},\dots \}$, are used to add positional information about the input patches to the transformer. We input these position-aware tokens into the transformer to produce a “transformer output.” We created the output image by reshaping the “transformer output” to the shape of the original image.

5.1.

Season Transfer

In Fig. 9, we show an example of season transfer for the Alps dataset. Real winter and real summer images are shown in columns (a) and (c), respectively, whereas the synthetic generated images are displayed in columns (b) and (d). We report the color image obtained by putting together the RGB bands (first row) and the single-image bands (rows 2 to 5). As it can be seen, the synthetic images produced by the pix2pix model approximate very well the real images, and in any case, they provide a realistic view of the framed region in the target season. A similar example for the Scandinavian dataset is shown in Fig. 10. Even in this case, the synthetic images are very close to the real ones.

Fig. 9

Alps season transfer example: (a) real winter, (b) generated winter, (c) real summer, and (d) generated summer.

Fig. 10

Scandinavian season transfer example: (a) real winter, (b) generated winter, (c) real summer, and (d) generated summer.

5.2.

Land Cover Transfer

In Fig. 11, we show an example of a land cover transfer from barren to vegetation and vice versa. We notice that the generated vegetation bands are darker than the input barren images and consistent with the real vegetation. In the same way, the generated barren image is lighter than the vegetation input as it should be for a real barren terrain. To give an objective measure of the effectiveness of the land cover transfer (in the absence of ground truth images), we utilized the NDVI index that is usually adopted to estimate the vegetation content of satellite multispectral images and defined as

Fig. 11

Land cover transfer examples: (a) real barren, (b) generated vegetation, (c) real vegetation, and (d) generated barren.

Table 2

Percentage of pixels classified into four terrain classes based on NDVI.

5.3.

IGPT

In Fig. 12, we show some images containing splices generated by the iGPT model. Figure 12(a) shows the original images, (b) the replacement masks (spliced region), and (c) contains the manipulated images. We can note that the model can generate splices, which blend well into the images. The spliced images contain areas with different climates, vegetations, and urbanization levels. For example, the top urban region in Fig. 13(a) has been synthetically generated, whereas the bottom green vegetation part belongs to the original image. In the same figure, the bottom vegetation part in Fig. 13(b) corresponds to a synthetic region, whereas the surrounding barren pixels are part of the original image.

Fig. 12

Example images generated by the iGPT architecture: (a) the original world image, (b) the spliced regions, and (c) the manipulated images.

Fig. 13

(a), (b) Generated image parts using iGPT.

5.4.

Vision Transformer

In Fig. 14, we show some examples of the spliced images generated by the vision transformer. Figure 14(a) contains the original images, (b) contains the replacement masks (spliced region), and (c) contains the manipulated images. We generated the spliced regions as in Sec. 5.3. As it can be seen, the vision transformer generates realistic spliced regions. The spliced images contain areas with a different climate including desert, Mediterranean, continental, tundra, and different levels of vegetation. In addition, the spliced regions blend well into the surrounding areas. For example, the bottom forest area in Fig. 15(a) has been synthetically generated, whereas all the other green vegetation parts belong to the original image. In the same figure, the top desert part in Fig. 15(b) corresponds to a synthetic region, whereas the other surrounding desert pixels are original.

Fig. 14

Examples of images generated by the vision transformer architecture: (a) the original world image, (b) the spliced regions, and (c) the manipulated images.

Fig. 15

(a), (b) Generated image regions using the vision transformer.

We noticed several differences in the splices generated by iGPT and vision transformer. iGPT tends to generated more diverse objects. For example, in Fig. 13(a), the original image was mainly vegetation and rural areas while the generated spliced region is an urban area. IGPT was able to generate splices with a pixel level detail for varying spliced regions, however, it took a long time (5 to 30 min) to generate these splices. The vision transformer tends to generate splices very similar to their surroundings. For example, in Fig. 15(a), the original image was a rural housing area, whereas the generated splices were vegetation, which is consistent with the rest of the image that is mainly occupied by vegetation. The vision transformer sometime had some difficulties to generate very detailed spliced regions, however, creating a spliced image took several seconds in contrast to the iGPT long generation time.