aidp-bucket-mapping — wire up s3:// ↔ oci://

Several places in the migrator look up s3://<bucket>/<path> and rewrite to oci://<bucket>@<namespace>/<path>:

• aidp-migrate-catalog rewrites external-table LOCATION clauses.
• aidp-migrate-job Pass-1 rewrites spark.read.parquet("s3://...") literals in notebook cells.
• aidp-check-data probes paths via the same translation.

All consult the same bucket-mapping config.

When to use

• Setting up the migrator on a new workstation / new tenancy combo.
• Any time a tool reports S3 bucket "<name>" not found in OCI bucket mapping. Known buckets: [...].
• After provisioning a new OCI Object Storage bucket that mirrors a Databricks-side S3 bucket.

The config file

The migrator loads bucket mappings via the load_bucket_mapping() helper. The customer supplies a JSON file with this shape (file path is configurable via --bucket-mapping):

{
  "buckets": {
    "<s3-bucket-name>": {
      "oci_bucket": "<oci-bucket-name>",
      "oci_namespace": "<oci-namespace>",
      "notes": "optional human note"
    },
    "<another-s3-bucket>": {
      "oci_bucket": "<another-oci-bucket>",
      "oci_namespace": "<oci-namespace>"
    }
  },
  "default_namespace": "<oci-namespace>",
  "default_region": "<oci-region>"
}

Save the file to a path the user controls (gitignored — it contains tenancy-specific identifiers). Pass via --bucket-mapping <path> to every migrator entrypoint.

Building the mapping for a new tenancy

If the user is doing this for the first time:

1. List the source S3 buckets referenced in the Databricks workspace. Quick way:

   # On the migrator repo:
   grep -roE 's3://[a-z0-9.-]+' <source-databricks-checkout>/ | sort -u
   

   Or use the migrator's prep helper if available.

2. For each, identify the target OCI bucket + namespace. Either:

   • The user provisions matching OCI buckets (recommended for big migrations — preserves bucket names).
   • The user routes everything into a single shared OCI bucket with prefix isolation.

3. Find the OCI namespace. Each OCI tenancy has ONE namespace per region — find it via:

   oci os ns get --profile <profile>
   # returns: {"data": "<your-namespace>"}
   

4. Write the JSON and place it at config/bucket_mapping.json (or wherever your team stores secrets). Make sure it's gitignored.

5. Test the mapping by re-running aidp-check-data with --bucket-mapping <path>. Any MISSING entries with s3://... paths now resolve to oci://... and probe correctly.

When the mapping should fail FAST (vs warn)

Behavior contract:

• migrate_catalog.py REJECTS unknown buckets when --catalog-manifest is explicit (data-correctness gate).
• job_migrate.py Pass-1 surfaces unknown buckets in the cell-fix Claude context so the model can either route to a synthetic stub OR ask the user.
• check_data_availability.py reports unknown-bucket as a hard MISSING row.

If you see WARNINGS but the migration continues, that's usually safe — the rewriter passed the path through unchanged. Confirm the consumer notebook either no longer reads that path OR has been adapted.

Common mistakes

After this

• Re-run aidp-check-data — any MISSING rows that were due to bucket-map issues should now be OK.
• Proceed to aidp-migrate-job / aidp-migrate-catalog.