A lawyer for an American who has spent more than four years imprisoned in Cuba said Monday that his client cannot take life in prison much longer and has said his goodbyes to his wife and a daughter.

Alan Gross was arrested in Cuba in 2009 while working covertly in the Communist-run country to set up Internet access. His attorney, Scott Gilbert, said in a statement Monday that his client “has withdrawn” and told him “life in prison is not a life worth living.” Gross has previously said through his lawyer that his 65th birthday, which took place in May, would be the last one that he “celebrates in Havana, one way or the other.”

Gross said goodbye to his wife and youngest daughter during a recent visit. Gross, who lived in Maryland before his arrest, had previously told his two daughters not to come see him in prison.

Gross has stopped exercising and his health is not good, said Gilbert, who plans to visit his client this week. His hips are failing and he has lost most of the vision in his right eye. Gross’ “emotional deterioration has been severe,” Gilbert said, particularly following the death in June of his 92-year-old mother.

Gross and his mother talked frequently by phone, and when Gross went on a nine-day hunger strike in April it was his mother who persuaded him to end it. Gross had asked the Cuban government to be able to return to the United States for her funeral, but the request was denied.

Gross’ wife, Judy Gross, said in Monday’s statement that she has never seen her husband in such bad shape during the time he has spent in prison. She wrote that “his decision to say goodbye to us was wrenching.”

Gross is also now refusing to see officials from the U.S. Interests Section in Havana, which Washington maintains instead of an embassy since the two countries do not have formal diplomatic relations.

At the time Gross was arrested he was working as a subcontractor for the U.S. government’s U.S. Agency for International Development, which promotes democracy on the island. Cuba considers USAID’s programs illegal attempts by the U.S. to undermine its government, and Gross was tried and sentenced to 15 years in prison.

Also on Monday, Gross’ lawyer released a letter from a group of 300 rabbis to President Barack Obama urging him to take action to secure Gross’ release. Gross, who is Jewish, was working with Cuba’s small Jewish community on Internet access when he was arrested.