By Alfonso Chardy
Herald Washington Bureau

WASHINGTON -- The Fidel Castro government has renounced an agreement with the United States in which Cuba helped crack down on U.S. drug smugglers, Cuban diplomatic sources said Tuesday.

They said that Cuban Foreign Ministry officials delivered a note to the American Interest Section in Havana late last week renouncing the mutual Coast Guard agreement reached during the Carter Administration.

Under the accord, born out of two sets of talks between U.S. and Cuban officials in 1978 and 1979 U.S. and Cuban coast guard crews agreed to cooperate in air and sea rescue operations and the Cubans also agreed to help the Americans m efforts against drug smugglers.

"We did this as a consequence of the American decision to further restrict travel to Cuba," said a Cuban diplomat of the Cuban Interests Section in Washington.

He said the move was the first official reaction of the Castro government to the April 19 decision by the Treasury Department to ban U.S. tourists and businessmen from visiting Cuba. Travel to Cuba had been allowed as a result of President Carter's willingness to ease relations with Castro

Under the new U.S. restrictions, which go into effect May 15, American officials, journalists and scholars and Cuban exiles can still travel to Cuba.

Administration officials said the narcotics segment of the Coast Guard accord became a controversial part of U.S.-Cuba relations. Under the agreement, dozens of Americans in boats and planes were arrested by Cuban authorities in the last five years and charged either with attempting to smuggle drugs or with violating Cuban air space or territorial waters.

In some cases, some of the Americans were held for years in Cuban jails. More than 30 such Americans were unilaterally released by the Cuban government in the fall of 1980 in what was seen then as a gesture of Cuban goodwill to try to help the sagging electoral prospects of Carter.

But Cuba refused to release a half-dozen Americans of Cuban origin on grounds that Havana did not recognize their American citizenship.

Aside from the controversy caused by the arrests, Reagan Administration officials say the accord. which is not a written agreement, was "of limited value." But Latin American diplomatic analysts say the renunciation of the accord marks a further deterioration in the already tense atmosphere between the United States and Cuba.

They add, however, that the step is actually a low-level reaction on the part of the Castro regime, and perhaps even an indication that the Cuban leader is still hopeful that he can somehow improve relations with the Reagan Administration.

"Obviously, Fidel still wants to continue talking to the Americans, otherwise the reaction would have been harsher," said a Latin American diplomat close to the Cubans.