RO6870810 (formerly TEN-010) is a small molecule, bromodomain and extra-terminal (BET) bromodomain inhibitor. This study is designed to characterize the safety, tolerability, and pharmacokinetics of RO6870810 monotherapy in participants with relapsed/refractory acute myeloid leukemia (RR-AML) and hypomethylating agent (HMA)-refractory myelodysplastic syndrome (MDS). The study will consist of a Screening Period, Treatment Period, and Post-Treatment Period. A standard 3+3 design will be used in which successive cohorts of three or more participants with RR-AML or HMA-refractory MDS will be treated at escalating doses until a maximum tolerated dose (MTD) is identified. Up to 51 adult participants with AML or MDS will be enrolled in the study.

Inclusion Criteria:

• RR-AML

• Relapsed/refractory MDS

• Participants with a history of allogeneic stem cell transplant are eligible for study participation provided the following eligibility criteria are met:

  1. Transplant was more than (>) 100 days prior to study enrollment
  2. Participant has not taken immunosuppressive medications for at least 2 weeks
  3. No signs or symptoms of graft versus host disease other than Grade 1 skin involvement
  4. No active infection

• Eastern Cooperative Oncology Group Performance Status score equal to or less than (<=) 2

• Life expectancy of at least 2 months

• Disease-free of active second/secondary or prior malignancies for equal to or more than (>=) 1 year with the exception of currently treated basal cell or squamous cell carcinoma of the skin, or carcinoma in-situ of the cervix or breast

• Adequate hematological, renal, hepatic and coagulation laboratory test results

• Women of childbearing potential and men must agree to use adequate contraception from 28 days prior to the first dose of the study drug, during the entire Treatment Period, and for at least 28 days after the last dose of the study drug

Exclusion Criteria:

• New York Heart Association Class III or IV cardiac disease, myocardial infarction within the past 6 months, unstable arrhythmia
• Have Fridericia-corrected QT interval > 470 milliseconds (msec) (female) or > 450 msec (male), or history of congenital long QT syndrome
• Uncontrolled bacterial, viral, or fungal infections
• Known clinically important respiratory impairment
• Positive for human immunodeficiency virus, hepatitis B surface antigen, or hepatitis C antibodies
• History of major organ transplant
• Symptomatic central nervous system disease, malignancy, or metastasis
• Pregnant or nursing
• Concomitant chemotherapy, radiation therapy, immunotherapy, biologic therapy, or hormonal therapy
• Treatment with surgery or chemotherapy within 21 days prior to study entry
• Prior treatment with small molecule bromodomain and extra terminal family inhibitor
• Radiation for symptomatic lesions within 14 days of study enrollment
• Active substance abuse