Hi Thisgame,
welcome to the fold

. I see Athena's been having her way with you

. Not to worry, we all went through this.
A couple of things:
1) get the latest patch from this thread:
New official 1.14 patch (the latest is always linked in the first post) so that everything (almost everything) is WAD (Working As Designed), but be advised, there will be a new (possibly last) patch out in a few days
Pocus, when can we expect the legacy patch?.
2) Athena (the game AI) does so very much love to put units/stacks in your back yard to keep you busy, either to draw-off pressure from another region, or to distract you while planning a thrust elsewhere. My experience (playing the Union) is it's best to be patient. If they're behind your lines, they are also cutoff from their supply base. That means, they can't hold out there for too long.
If the enemy stack is in a city (this is mostly what Athena does) and you can place a stack of your own of at least equal strength in the same region, the enemy stack will become besieged. Basically, unless the enemy now sorties out of the city to attack you, it's trapped, it receives no supply (unless the city has an un-blockaded harbor) and each turn a Siege Resolution is executed, which can result in a Breach being made and the enemy taking 'hits'. Breaches reduce the protection the enemy unit gets in a city/fortification (eventually to 0). (If you click on one of your units/stacks, it appears at the bottom of your display in the 'stack display', then click on the NATO symbol in the upper right corner of one of those units, the 'unit display' opens to the right of the 'stack display'. The little men at the top of the 'unit display' denotes how many hits a unit can take)
That may sound like the up to 10 hits you may make per turn through besieging would take forever to reduce the enemy stack, and actually it would, but during the turn resolution Athena will automatically trade those hits against available supply in the enemy stack. The enemy stacks own supply usage plus the hits you are causing will slowly but surely reduce the enemy's supply to 0, at which point the enemy stack starts loosing 'hits' from all the units in the stack due to being out-of-supply.
Also the enemy stack will start to get disorganize (lose cohesion), especially if it sorties out to fight, which it can only slowly recover while being besieged. The stack will eventually start to crumble to the point where your besieging units can risk an assault.
If you have enough units to rotate in and out of the region, you can also attempt to assault every couple of turns even if you know you will lose, because the bashed up units you just assaulted with can be relieved by fresh ones. Repeat until success.
3) About those pesky cavalry raiders, the best solution is to garrison everything of value in your hinterland. One militia regiment in a city is enough to prevent them from attacking a depot. Chase them with your own cavalry. Don't just move to the region, but drag your cavalry unit on top of the enemy unit until the enemy unit brightens up. Now drop your unit. You will now see an icon on top of your unit (the real one, not the dark one denoting to where you are moving that unit) that looks like three soldiers with a pair of scissors on top of them. This means that your unit is intercepting the enemy unit and will actually try to follow them. It doesn't work that often when intercepting cavalry, but you can also continue plotting with your attacking units by dragging them to a further region to where you think the enemy raiders might move.
I've never played the CSA, as it seams that you are doing, so my suggestions my have to be modified for your situation. The Union player generally always has a lot more units, though they are almost always weaker (especially the leaders) than the CSA, especially early in the war. You may be able to put a stack together with strong units and leaders with good strategy and assault values and just smash a Union stack right from the get-go of them being in your back yard. You'll have to evaluate per situation.
Hope this helps, and don't despair. You'll get a hang of it yet.