The drug that will be investigated in the trial is an antibody, GEN3018. Since this is the first trial of GEN3018 in humans, the main purpose is to evaluate safety. In addition to safety, the trial will determine the recommended GEN3018 dose(s) to be tested in a larger group of participants and assess preliminary anti-tumor activity of GEN3018. GEN3018 will be studied in refractory (resistant to treatment) or relapsed (disease has returned) acute myeloid leukemia (also known as R/R AML) and refractory or relapsed higher-risk myelodysplastic syndrome (also known as R/R HR-MDS). The trial consists of 2 parts:

1. Part 1 Dose Escalation will test increasing doses of GEN3018 to identify a safe dose level to be tested in the next part
2. Part 2 Dose Refinement will further test the GEN3018 dose(s) determined from the Dose Escalation.

Up to 78 participants may be treated in this trial (up to 60 participants in Part 1; up to 18 participants in Part 2).

For an individual participant in the trial, the estimated treatment duration will be up to 1 year. Participation in the trial will require regular scheduled visits to the site. At site visits, there will be various tests (such as blood draws) to monitor whether the treatment is safe and effective. Participants will also be contacted every 3 months after treatment ends to monitor how they are doing.

All participants in the trial will receive active drug (ie, GEN3018); no one will be given placebo.

This is a FIH, Phase 1, open-label, multicenter trial in participants with R/R AML or R/R HR-MDS, to evaluate the safety, tolerability, pharmacokinetics (PK), immunogenicity, pharmacodynamics, and preliminary anti-tumor activity of GEN3018. The trial will be conducted in 2 parts: Dose Escalation (Part 1) and Dose Refinement (Part 2).

Key Inclusion Criteria:

All Participants:

• Be at least 18 years of age at the time of signing informed consent form (ICF).
• Participant's life expectancy at screening is judged to be at least 3 months.
• Must have fresh bone marrow samples collected at screening.
• Bone marrow (BM) blasts ≥ 5% at screening.
• Eastern Cooperative Oncology Group (ECOG) performance status (PS) score of ≤ 2.
• Has acceptable laboratory test results during the screening period

Participants with R/R AML:

• Relapsed or refractory AML, either de novo or secondary, and must have failed all conventional therapies.
• Relapsed or refractory to at least one prior line of therapy.

Participants with R/R HR-MDS:

• Diagnosed with high- or very-high risk MDS according to International Prognostic Scoring System (IPSS-R) (score of > 4.5 ie, high or very high) or World Health Organization (WHO) 2022 classification (ie, MDS-IB1 or MDS-IB2).
• Refractory or relapsed after hypomethylating agents (HMAs) (such as azacitidine or decitabine).

Key Exclusion Criteria:

All Participants:

• Diagnosis of acute promyelocytic leukemia (APL).
• Presence of extramedullary AML at screening.
• Prior autologous or allogenic hematopoietic stem cell transplant (HSCT) within 3 months prior to initiation of trial treatment.
• Active graft-versus-host disease.
• History of severe immune-related adverse events.
• Treatment with anti-cancer agent (eg, small molecule, antibody, chemotherapy, radiation therapy), or major surgery within 2 weeks prior to the first dose of GEN3018.

Other protocol-defined Inclusion and Exclusion criteria may apply.

Randomization will not be used in this trial for Part 1 Dose Escalation, but may be used for Part 2 Dose Refinement.