Hi,
The differences are coming due to direct GL postings to stock account. As per SAP standard, the stock accounts to be posted automatically only. Please read the note 968812 , which explains the particular issue in detail.
Physical inventory is not for stock value check. It is actually for stock quantity correction so that the stock value automatically get corrected.
Hope its helpful.
Regards,
AKPT