This app creates two types of transfers using the Transfer Service tool.
These samples are used on the following documentation pages:
- https://cloud.google.com/storage/transfer/create-client
- https://cloud.google.com/storage/transfer/create-manage-transfer-program
-
Set up a project on Google Cloud Console.
-
Go to the Google Cloud Console and create or select your project. You will need the project ID later.
-
Enable the Google Storage Transfer API in the Google Cloud Console.
-
Within Developers Console, select APIs & auth > Credentials.
-
Select Add credentials > Service account > JSON key.
-
Set the environment variable
GOOGLE_APPLICATION_CREDENTIALSto point to your JSON key. -
Add the Storage Transfer service account as an editor of your project.
-
To get the email address used for the service account, execute the googleServiceAccounts.get REST method. There should be a "Try It" section on that page, otherwise execute it in the APIs Explorer.
It should output an email address like:
storage-transfer-1234567890@partnercontent.gserviceaccount.com -
Add this as a member and select the Project -> Editor permission on the Google Cloud Console IAM and Admin page.
-
Set up gcloud for application default credentials.
-
gcloud components update -
gcloud init
Creating a one-time transfer from Amazon S3 to Google Cloud Storage.
- Set up data sink.
- Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
- Set up data source.
- Go to AWS Management Console and create a bucket.
- Under Security Credentials, create an IAM User with access to the bucket.
- Create an Access Key for the user. Note the Access Key ID and Secret Access Key.
- Set the
AWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEYenvironment variables. - Compile the package with
mvn compile
- Run the transfer job with
mvn exec:java \ -Dexec.mainClass="com.google.cloud.storage.storagetransfer.samples.AwsRequester" \ -DprojectId=your-google-cloud-project-id \ -DjobDescription="Sample transfer job from S3 to GCS." \ -DawsSourceBucket=your-s3-bucket-name \ -DgcsSinkBucket=your-gcs-bucket-name- Note the job ID in the returned Transfer Job.
Creating a daily transfer from a standard Cloud Storage bucket to a Cloud Storage Nearline bucket for files untouched for 30 days.
- Set up data sink.
- Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
- Select Nearline for Storage Class.
- Set up data source.
- Go to the Developers Console and create a bucket under Cloud Storage > Storage Browser.
- In NearlineRequester.java, fill in the user-provided constants.
- Run with
mvn compileandmvn exec:java -Dexec.mainClass="com.google.cloud.storage.storagetransfer.samples.NearlineRequester" - Note the job ID in the returned Transfer Job.
- In RequestChecker.java, fill in the user-provided constants. Use the Job Name you recorded earlier.
- Run with
mvn compileandmvn exec:java -Dexec.mainClass="com.google.cloud.storage.storagetransfer.samples.RequestChecker"