Society at Roxbury, had noted this happy condition, and the hopeful outlook of the church. It extended to Mr. Lee a call to become rector of All Saints, Worcester (March 3, 1846.) It felt that that Church needed a man of Mr. Lee's gifts. But his parishioners remembered too well the lean years through which they had recently passed and felt they could not permit their leader to leave them at this time. We record especially the words of Mr. Diah Allin. Mr. Allin played a conspicuous part in the early Christ Church history. His children were among the first to be baptized in the Armory Hill Chapel and he was one of the first vestrymen there. In later years, Bishop Lee writing to Mr. Allin called him "My sincere and devoted friend who today is prominently connected with the scenes that are passing in review before me." Regarding Mr. Lee's going to Worcester, Mr. Allin wrote to the wardens and vestrymen, earnestly asserting that the Board of Missions did not realize how keenly Mr. Lee is needed here, even if all the debts were paid and the church was so well established; he suggested a larger salary, and added, " . . . . I fear we should find our congregation dwindling away, and that we should very soon find ourselves in a most unpleasant and unhappy predicament." Mr. Lee deeply appreciative of this well deserved attitude decided to decline this call. He wrote, March 14, 1846, " . . . . I am cheered and rejoiced to know that I have the affection and confidence of those to whom it is my privilege to minister in holy things."
Bishop Alexander Viets Griswold died during Mr. Lee's rectorship, and a Memorial Service in his honor was held in Christ Church. "The Church was shrouded in mourning; all day at the usual hours the bell tolled,—and in the afternoon a funeral service was performed; after which, as the people were retiring, the age of the man old and full of years was announced by the solemn peal."
Two years later, Mr. Lee was called to Rochester, N.Y. This time, he deemed it wise to accept the call (Nov. 18,1847), although he wrote of his feelings later.