Avelumab, Utomilumab, Rituximab, Ibrutinib, and Combination Chemotherapy in Treating Patients With Relapsed or Refractory Diffuse Large B-Cell Lymphoma or Mantle Cell Lymphoma

All patients and/or their parents or legal guardians must have the ability to understand and the willingness to sign a written informed consent

Voluntary written informed consent must be obtained before performance of any study-related procedure not part of normal medical care, with the understanding that consent may be withdrawn by the subject at any time without prejudice to future medical care

Weight over 40 kilograms (kg)

Life expectancy of greater than 3 months

Cohort #1: histologically confirmed CD20-positive, relapsed or refractory DLBCL, including de novo and transformed DLBCL (from follicular or marginal zone lymphoma); this includes patients with DLBCL who are found to have small cell infiltration of the bone marrow or other diagnostic material (representing a discordant lymphoma)

Cohort #1: patients must be either refractory to or relapsed after up to 2 lines of prior therapy

Cohort #2: histologic confirmation of relapsed or relapsed/refractory MCL confirmed by presence of cyclin D1 by immunohistochemistry (IHC) or fluorescence in situ hybridization (FISH)

Patients must have measurable disease \> 1.5 cm evidenced by computed tomography (CT) scan of the neck/chest/abdomen (abd)/pelvis or CT/positron emission tomography (PET) scans

Cohort #2: patients must be either refractory to or relapsed after at least 1 line of prior therapy

Eastern Cooperative Oncology Group (ECOG) performance score of 0-2

Cohort #1: received only frontline CD20-directed immunotherapy with anthracycline- or anthracenedione-based multi-agent chemotherapy for patients with DLBCL; monotherapy rituximab or other CD20-directed immunotherapy prior to frontline chemotherapy, as maintenance therapy, and radiotherapy in a limited field or as a part of the frontline treatment plan are permitted; last treatment dose should be 3 weeks before start of study treatment

Cohort #1: considered eligible for high-dose chemotherapy followed by autologous stem cell transplantation (ASCT)

Cohort #2: patients with MCL with prior allogeneic hematopoietic stem cell transplant, minimum 6 months after transplant, not on immunosuppression, and without prior or active graft versus host disease (GVHD), are allowed

Cohort #2: prior treatment with ibrutinib is allowed; patients should not have received any anti-lymphoma therapy within 3 weeks from start of study treatment, with the exception of ibrutinib

Absolute neutrophil count (ANC) \>= 1,000/mm\^3 (performed within 14 days prior to day 1 of protocol therapy)

* Filgrastim can be given prior to enrollment to achieve target ANC \>= 1000/uL

Platelets \>= 50,000/mm\^3 for MCL cohort and platelets \>= 75,000/mm\^3 for DLBCL cohort (performed within 14 days prior to day 1 of protocol therapy)

* NOTE: platelet transfusions and packed red blood cell transfusion can be given prior to enrollment to achieve a target platelet (Plt) \>= 50,000/uL for MCL and \>= 75,000/uL for DLBCL and hemoglobin of \>= 8.5 g/dL

Hemoglobin \>= 8.5 g/dL (performed within 14 days prior to day 1 of protocol therapy)

Total bilirubin within 1.5 x the upper limit of normal institutional limits; patients with elevation of unconjugated bilirubin alone, as in Gilbert's disease, are eligible (performed within 14 days prior to day 1 of protocol therapy)

Aspartate aminotransferase (AST) =\< 3 x upper limit of normal (ULN) unless demonstrated lymphoma involvement of the liver (performed within 14 days prior to day 1 of protocol therapy)

Alanine aminotransferase (ALT) =\< 2.5 x ULN unless demonstrated lymphoma involvement of the liver (performed within 14 days prior to day 1 of protocol therapy)

Creatinine clearance \>= 50 mL/min for Cohort #1 (DLBCL) and \>= 30 mL/min for Cohort #2 (MCL) per the Cockcroft-Gault formula (performed within 14 days prior to day 1 of protocol therapy)

If not receiving anticoagulants: international normalized ratio (INR) OR prothrombin (PT) =\< 1.5 x ULN (performed within 14 days prior to day 1 of protocol therapy)

* If on anticoagulant therapy: PT must be within therapeutic range of intended use of anticoagulants

If not receiving anticoagulants: activated partial thromboplastin time (aPTT) =\< 1.5 x ULN (performed within 14 days prior to day 1 of protocol therapy)

* If on anticoagulant therapy: aPTT must be within therapeutic range of intended use of anticoagulants

Female of childbearing potential: negative urine or serum pregnancy test, performed within 14 days prior to day 1 of protocol therapy

* If the urine test is positive or cannot be confirmed as negative, a serum pregnancy test will be required

Cardiac function (12 lead- electrocardiogram \[ECG\] versus \[vs\] non-12 lead ECG), performed within 14 days prior to day 1 of protocol therapy

Female subjects must be either post-menopausal, surgically sterilized, or willing to use an acceptable method of birth control (i.e. a hormonal contraceptive, intra-uterine device, diaphragm with spermicide, condom with spermicide, or abstinence) beginning prior to study entry, for the duration of the study, and for six months following last dose of avelumab/utomilumab; should a woman become pregnant or suspect that she is pregnant while participating on the trial, she should inform her treating physician immediately

Male subjects must agree to use an acceptable method of contraception beginning prior to study entry, for the duration of the study, and for six months following last dose of avelumab/utomilumab

CRITERIA SPECIFIC FOR COHORT #2 (MCL): Currently active, clinically significant hepatic impairment Child-Pugh class B or C according to the Child Pugh classification

CRITERIA SPECIFIC FOR COHORT #2 (MCL): Unable to swallow capsules or malabsorption syndrome, disease significantly affecting gastrointestinal function, or resection of the stomach or small bowel, symptomatic inflammatory bowel disease or ulcerative colitis, or partial or complete bowel obstruction

CRITERIA SPECIFIC FOR COHORT #2 (MCL): Known bleeding disorders (e.g., von Willebrand's disease) or hemophilia

CRITERIA SPECIFIC FOR COHORT #2 (MCL): Major surgery within 4 weeks of first dose of study drug

Active GVHD or on immunosuppressive medication for GVHD (applies to cohort #2)

Recent infection requiring intravenous anti-infective treatment that was completed =\< 14 days before enrollment; active infection requiring systemic therapy

Persisting toxicity related to prior therapy (National Cancer Institute \[NCI\] Common Terminology Criteria for Adverse Events \[CTCAE\] version \[v\]. 4.03 grade \> 1); however, alopecia, sensory neuropathy grade =\< 2, or other grade =\< 2 not constituting a safety risk based on investigator's judgement are acceptable

Hypersensitivity to chemotherapy or history of allergic reactions attributed to compounds of similar chemical or biologic composition to rituximab, ibrutinib, ICE, Ave, or Uto

Patients should not have any uncontrolled illness including ongoing or active infection

Any life-threatening illness, medical condition, or organ system dysfunction that, in the investigator's opinion, could compromise the subject's safety or put the study outcomes at undue risk

Myocardial infarction within 6 months prior to enrollment or New York Heart Association (NYHA) class III or IV heart failure, uncontrolled angina, severe uncontrolled ventricular arrhythmias, or electrocardiographic evidence of acute ischemia or active conduction system abnormalities; prior to study entry, any ECG abnormality at screening has to be documented by the Investigator as not medically relevant

Clinically significant (i.e., active) cardiovascular disease: cerebral vascular accident/stroke (\< 6 months prior to enrollment), myocardial infarction (\< 6 months prior to enrollment), unstable angina, congestive heart failure (\>= New York Heart Association classification class II), or serious cardiac arrhythmia requiring medication

Diagnosed or treated for another malignancy within 3 years of enrollment, with the exception of complete resection of basal cell carcinoma or squamous cell carcinoma of the skin, an in situ malignancy, or low-risk prostate cancer after curative therapy

Patients with active central nervous system (CNS) disease or history of brain metastases are excluded from study

Active human immunodeficiency virus (HIV) or hepatitis C virus (HCV) or hepatitis B virus (HBV); subjects who have an undetectable HIV viral load with CD4 \>= 200 and are on highly active antiretroviral therapy (HAART) medication are allowed; subjects who are positive for hepatitis B core antibody or hepatitis B surface antigen must have a negative polymerase chain reaction (PCR) result before enrollment; those who are PCR positive will be excluded; patients who have had hepatitis C but have finished treatment and are PCR negative will be allowed; (testing to be done only in patients suspected of having infections or exposures)

Active autoimmune disease that might deteriorate when receiving an immuno-stimulatory agent; patients with diabetes type I, vitiligo, psoriasis, or hypo- or hyperthyroid diseases not requiring immunosuppressive treatment are eligible

Other severe acute or chronic medical conditions including immune colitis, inflammatory bowel disease, immune pneumonitis, pulmonary fibrosis or psychiatric conditions including recent (within the past year) or active suicidal ideation or behavior; or laboratory abnormalities that may increase the risk associated with study participation or study treatment administration or may interfere with the interpretation of study results and, in the judgment of the investigator, would make the patient inappropriate for entry into this study

Pregnant women are excluded from this study; breastfeeding should be discontinued

Other active malignancy

Any other condition that would, in the investigator's judgement, contraindicate the patient's participation in the clinical study due to safety concerns with clinical study procedures

Prospective participants who, in the opinion of the investigator, may not be able to comply with all study procedures (including compliance issues related to feasibility/logistics)

Unable to understand the purpose and risks of the study and to provide a signed and dated informed consent form (ICF) and authorization to use protected health information (in accordance with national and local subject privacy regulations)

What this study is about