I'd suggest unscrewing the filter bowl and pour it into a tub to see if there's anything floating around in there, and inspect the filter element again
The problem might be in the fuel tank.
I've read about issues before, mainly in boats mind you, caused by diesel bug.
Do you know if there's a percentage of biodiesel in what you buy from the fuel station?

Here's an article on it.
12 diesel bug treatments tested - Practical Boat Owner

It's just when you seemed to cure it by changing the filter, but it's come back again, it makes me think it's possibly something in the tank.