Turkey accuses the cleric, Fethullah Gulen, of orchestrating last July's failed coup.
In a video interview, Mr Woolsey told the Wall Street Journal he was present at a discussion about removal methods beyond the legal extradition process.
Mr Flynn disputes Mr Woolsey's account.
The meeting took place last September at a New York Hotel.
Those present included Mr Flynn, then an adviser on national security to the Trump election campaign, the son-in-law of Turkish President Recep Tayyip Erdogan, and Turkish Foreign Minister Mevlut Cavusoglu, according to the WSJ (paywall).
Mr Woolsey, then also advising the Trump campaign, admitted he might not have heard "some kind of caveat" because he arrived late for the meeting.
But he told the WSJ: "There was serious discussion of finding some way to move Mr Gulen out of the US to Turkey.
"You might call it brainstorming. But it was brainstorming about a very serious matter that would pretty clearly be a violation of law."
He went on: "It was a serious and troubling discussion but it did not, repeat not, in my portion of being in the room, rise to a level of being a specific plan to undertake a felonious act."
In a later interview with CNN, Mr Woolsey called the meeting "suspicious" and "concerning", saying: "I felt I needed to say something to somebody, but was it a clear plot that they were going to seize him? No."
A spokesman for Mr Flynn, whose consulting company Flynn Intel Group carried out work for the Turkish government, disputed Mr Woolsey's version of events.
"At no time did Gen Flynn discuss any illegal actions, non-judicial physical removal or any other such activities," he said.
The presence of Mr Gulen in the US has become a major irritant in relations between Washington and Ankara. Turkey has repeatedly called for him to be handed over.
Mr Gulen, who has lived in Pennsylvania since 1999, denies involvement in the failed coup.
Fethullah Gulen: Powerful but reclusive Turkish cleric
Michael Flynn: Former US national security adviser
Mr Flynn was forced to step down as national security adviser in mid-February after just a few weeks in post over allegations he discussed sanctions with Russia before Mr Trump took office.
Private US citizens are barred from conducting diplomacy. Mr Flynn admitted providing the vice-president with an incomplete account of his conversation with the Russian ambassador.

At an event in New York on Monday, Madonna, Beyonce, Kanye West, and 13 other performers announced they had become co-owners of Tidal.
The service launched last October, but was recently bought by rapper Jay Z.
It is hoping to compete with the likes of Spotify, Deezer and Google Play.
However Tidal offers 25 million music tracks, fewer than the 30 million offered by many rival services.
As well as a standard subscription for $9.99 a month, Tidal offers a "high fidelity" option for $19.99, which claims to deliver better sound quality.
Singer Alicia Keys spoke on behalf of the artists as they all lined up on stage at Skylight at Moynihan Station in Manhattan.
Describing the event as a "graduation", she said the artists hoped Tidal would alter musical evolution.
"So we come together before you on this day, March 30th, 2015, with one voice in unity in the hopes that today will be another one of those moments in time, a moment that will forever change the course of music history."
Their mission, she said, goes beyond commerce and technology.
"Our intent is to preserve music's importance in our lives,'' she said.
Notably, the majority of the company will be owned by artists, a symbolic move in a business where musicians often have little control over how their work is distributed and consumed.
Jay Z's strategy will include encouraging artists to lobby their labels to give Tidal new music a week in advance of other services, giving it a short window of exclusivity.
He told Billboard: "We didn't like the direction music was going and thought maybe we could get in and strike an honest blow.
"Will artists make more money? Even if it means less profit for our bottom line, absolutely. That's easy for us. We can do that. Less profit for our bottom line, more money for the artist; fantastic."
President Trump's former national security adviser, Michael Flynn, discussed removing from the US a Muslim cleric wanted by Turkey, according to ex-CIA director James Woolsey.
[NEXT_CONCEPT]
Some of the biggest names in entertainment have re-launched the music subscription service Tidal, which they are billing as the first artist-owned platform for music and video.